Skip to content

feat(ci): deterministic Odoo fixture + full verified/contradicted cov… - #753

Merged
Weegy merged 2 commits into
byte5ai:mainfrom
sneumannb5:feat/639-golden-set-eval-v2
Aug 20, 2026
Merged

feat(ci): deterministic Odoo fixture + full verified/contradicted cov…#753
Weegy merged 2 commits into
byte5ai:mainfrom
sneumannb5:feat/639-golden-set-eval-v2

Conversation

@sneumannb5

@sneumannb5 sneumannb5 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

What

Golden-set eval v2 (Closes #639). Inject FixtureOdooReader into DeterministicChecker so a corpus entry declares frozen Odoo records the real checker re-queries — closes Gap 1 (ERP-confirmed hard claim → verified → approved) and Gap 2 (ERP-refuted → blocked). expected.via asserts the path so an entry can't pass for the wrong reason. Gap 3 (full-turn eval) deferred.

Test plan

  • test/goldenRunner.test.ts (pure) — 33/33
  • test/goldenModel.test.ts (real pipeline, stub LLM) — 9/9
  • npm run lint && typecheck && test in middleware

Risk / blast radius

CI-only (test/golden/** + golden-eval.yml); no runtime/schema/API change, no new env-var.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@Weegy

Weegy commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Reviewed and refreshed — main merged in at 619ab089 (the branch was based on 825dc714, two merges behind, so its green was against a base that no longer exists).

Verified on the new head

  • goldenRunner.test.ts + goldenModel.test.ts42/42
  • full middleware suite — 6880 tests, 0 fail, 0 cancelled

The check I care about most passed

expected.via is a newly declared field, and the recurring defect in this repo is a field that nothing reads — it compiles, the tests pass, and the assertion is decorative. Not the case here: viaSatisfied is called at both decision sites, and the tests carry real negative cases (viaPass === false for the judge path, plus throws on an unknown expected.via).

Mutation-checked to be sure — forcing viaSatisfied to return true turns 2 tests red, and reverting returns 33/33. The path assertion is load-bearing, not decorative.

Cost, since this corpus is the one that just started actually running

I costed these evals recently, and got it wrong the first time by reading corpus size as call count. For anyone doing the same arithmetic here: the two new files are 2 entries, not 70 — the rest of those line counts is (excellent) commentary.

The one thing worth naming explicitly, because "no new env-var" in the description is true but is not the whole cost story: folding via into the resample guard means an entry with the right class but the wrong path now resamples, where before it would have stopped after one call. runEntry's own docstring already reasons this through correctly — green-on-first-sample costs one call, worst case stays bounded at maxRuns × corpus size — and the flake-tolerance argument for including via is sound. Flagging it only so the trade is visible in the PR, not just in the source.

No interaction with #748 (the temperature fix): the golden/verifier path sends no temperature, and DEFAULT_MODEL is claude-haiku-4-5, which accepts it regardless. Checked rather than assumed.

One nit

The description says the blast radius is test/golden/** + golden-eval.yml, but golden-eval.yml is not in the diff — the workflow is untouched. Worth correcting so a reader does not go looking for a CI change that is not there.

No blockers from me.

@Weegy
Weegy merged commit 1a4812d into byte5ai:main Aug 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(ci): golden-set eval v2 — deterministic-source + full-turn coverage (follow-up to #129)

2 participants